Low-Rank Based Algorithms for Rectification, Repetition Detection and De-noising in Urban Images

Repeated patterns (such as windows, tiles, balconies and doors) are prominent and significant features in urban scenes. Detection of the periodic structures is useful in many applications such as photorealistic 3D reconstruction, 2D-to-3D alignment, façade parsing, city modeling, classification, navigation, visualization in 3D map environments, shape completion, cinematography and 3D games, just to name a few. However it is a challenging task due to scene occlusion, varying illumination, pose variation and sensor noise. Therefore, detection of these repeated patterns becomes very important for city scene analysis. In this proposal, I first propose a method that attacks the problem of repeated patterns detection in a precise, efficient and automatic way, by combining traditional feature extraction followed by a Kronecker product low-rank modeling approach. Then I explain the limitations in the current method. In the last, I describe the future work that will be conducted to address the limitations. The proposed method is tailored for 2D images of building façades. The first step is to automatically select a representative texture within façade images using vanishing points and Harris corners. After rectifying the input images, I propose novel algorithms that extract repeated patterns by using Kronecker product based modeling that is based on a solid theoretical foundation. This approach is unique and has not ever been used for façade analysis. I have tested the algorithms in a large set of images, which includes building facades from Paris, Hong Kong and New York. Out of the 89 images I tested, only 4% resulted to failure detections. The results from the remaining 96% were very similar to the ground-truth. I manually labeled the ground-truth for all images. I overlaid my results with the ground-truth pixel by pixel and had exact matches for 91% of the pixels. There are still limitations though in the current model. I will continue to work on the following directions: improving the algorithm of estimating rank K (Sec. 3.2.1), and designing a better block partition algorithm for nested patterns (Sec. 3.2.4). I also plan to apply the Kronecker Product Model to 3D data sets. My recent studies have already produced some promising results on those proposed directions.